% scribe: Moorea Brega
% lastupdate: 22 November 2005
% lecture: 22
% references: Durrett, sections 3.1 and 4.2
% title: Stopping Times and Martingales
% keywords:
% end

% Put stuff here, Moorea

% indexing information (16 Jan 05)
%
% main: stopping times and martingales
% subjects: stopping times, martingales, gambler's ruin, Wald's identity, martingale transform, filtrations, optional times, martingale property
% local: Lecture 18
% Durrett: 3.1, 4.2
% Kallenberg: 7
% precedes: 
% follows: conditional expectation
% related:

\documentclass[12pt, letterpaper]{article} 
\include{macros}

\usepackage{floatflt}

%\def\qed{\ifmmode\Box\else$\Box$\fi}

\begin{document}

\lecture{22}{Stopping Times and Martingales}{Moorea Brega}{brega@stat}

These notes are a revision of old notes by Sridhar Machiraju.
References: \cite{durrett}, sections 3.1 and 4.2.

\section{Stopping Times}

Stopping times and martingales are both related to the idea of ``the information available at the present time.''  This is represented by an increasing family of $\sigma$-fields indexed by time --- in discrete time, this is just $\F_0 \subset \F_1 \subset \F_2 \subset \ldots$ --- called a \emph{filtration}, and $\F_n$ is thought of as the $\sigma$-field of ``events that are determined by time $n$.''  A sequence of random variables $\{X_n\}$ gives a filtration by $\F_n = \sigma(X_1,\ldots,X_n)$.  Conversely we say that a sequence of random variables $\{Y_n\}$ is \emph{adapted} to a filtration $\F_n$ if $Y_n \in \F_n$ for all $n \ge 1$.  We say that a sequence of random variables $\{Z_n\}$ is \emph{predictable} (with respect to $\{\F_n\}$) if $Z_n \in \F_{n-1}$, for all $n \ge 1$.

Let $\F = \{\F_n\}$ be a filtration.  A \emph{stopping time} $T$ is a 
random variable $T: \Omega \rightarrow \Z^+$ such that the event $\{T=n\}$ is $\F_n$-measurable:
$\forall~n < \infty$, $\{T = n\} \in \F_{n}$.   Two more equivalent conditions are:
$\{T \leq n\} \in \F_{n}$ and $\{T > n\} \in \F_{n}$.

Not all random variables are stopping times.  Consider, for example, the random variable $T$ defined by 

\[
T = \text{first index } i \le N \text{ s.t. } X_i = \max_{1 \le j \le N} X_j.
\]

Then, 

\[
(T = n) = (X_1 < X_n, \ldots, X_{n-1} < X_n, X_{n+1} \le X_n, \ldots, X_N \le X_n).
\]

Clearly $(T = n) \in \F_N$, but $(T=n) \notin \F_n$ in general.  The problem with this example is that the random variable $T$ requires us to know ``future" information.  As a humorous aside, another example of an adaptive strategy which is not a stopping time is the following rule for cooking toast:  cook toast until 10 seconds before it starts to smoke.

\begin{lemma}[Wald's Identity]
\rm Let $X, X_1 , X_2 , X_3 ,...$ be i.i.d. random variables with $\E|X_i| < \infty$ and
$T$ be a stopping time for $\{\F_n\}$ where $\F_n = \sigma (X_1 , X_2 ,..., X_n )$
with $\E(T) < \infty$. Let $S_n = X_1 + X_2 + ... + X_n$.
Then $\E S_{T} = \E X\E T$.
\end{lemma}

\begin{proof}
\begin{align*}
\E S_{T} = \E\left(\sum_{n=1}^{T}{X_n}\right) = \E \left( \sum_{n=1}^{\infty} X_n \1_{(T \ge n)} \right)  = \sum_{n=1}^{\infty}{\E(X_n \1_{(T\ge n)})} 
\end{align*}

Notice

\begin{align*}
 &(T \le n) \in \F_n \qquad n = 0, 1, 2, \ldots \\
\iff & (T > n ) \in \F_n \\
\iff & (T \ge n+1) \in \F_n \\
\iff & (T \ge n) \in \F_{n-1} \qquad n = 1,2, 3, \ldots
\end{align*}

We have $\1_{(T \ge n)} \in \F_{n-1}$ and $X_n$ independent of $\F_{n-1}$.  Thus, 


\[
\E S_T = \sum_{n=1}^{\infty}{\E X_n \E\1_{(T> n-1)}} = \E X \ \E\left( \sum_{n=1}^{\infty} \1_{(T \ge n)}\right) = \E X \E T.
\]
\end{proof}

It might seem as if $\E X_i$ being the same for all $i$ would suffice for Wald's identity to hold.  
More than this is needed, however.  We need $\E|X_i| < \infty$
to hold uniformly, without which the summation and integral (in the calculation of expected value
of $S_{T}$) cannot be exchanged. The following example illustrates this.

\begin{example}
Define $X_i$ as $\P(X_i=\pm 2^i) = \frac{1}{2}$. Let $T=\{$inf$~n: S_n > 1\}$.
Clearly, $\P(T=n) = \frac{1}{2}^n,~ \E T = 2 < \infty$ and $\E S_T\ge 1$. However,
$\E X_i=0$ which clearly violates Wald's identity.  
Note that here $\E|X_i|=2^i \toinf$, as $i \toinf$.
\end{example}

We now derive the classic \emph{Gambler's Ruin} formula using Wald's identity. The problem is 
that of a simple, symmetric random walk that starts at $X_0=0$, i.e.\ $X_i$ has a probability 
of $\frac{1}{2}$ for both $1$ and $-1$.  Let $a,b >0$ be two integers.  Define $T = \inf \{ n| S_n  = b \text{ or } S_n = -a \}$.  
Think about a gambler starting with a net profit of 0 and wondering about the chance she wins 
$b$ before experiencing a net loss of $a$.  
Now, $\P(S_T = b) = \P(T_b < T_{-a}) $ where $T_x= \inf\{n| S_n = x\}$.  Similarly, $\P(S_T = -a) = \P(t_{-a} < T_b)$.  Using Wald's identity with $EX = 0$, we have 

\[
\E (S_T) = b \P(T_b < T_{-a}) - a \P(T_{-a} < T_b) = 0.
\]

We also have 

\[
\P (T_b < T_{-a}) + \P(T_{-a} < T_b) = 1.
\]

Using these two equations, we find


\begin{align*}
\P(T_b < T_{-a}) = \frac{a}{a+b} \\
\P(T_{-a} < T_b) = \frac{b}{a+b}.
\end{align*}

%\[ \E(a+S_T) = b\P(a+S_T=b) = a + \E S_T = a \]
%by the definition of expectation, and Wald's identity.  Hence,
%the gambler earns $b$ before going broke with probability $\frac{a}{b}$.

\section{Martingales}

An $\{\F_n\}$-adapted sequence of random variables $\{M_n\}$ is a \emph{martingale} (MG) with respect to $\F_n$ if 
\begin{enumerate}
\item $\E|M_n| < \infty$, and
\item $\E(M_{n+1}|\F_n) = M_n \quad \forall n$.
\end{enumerate}
We define $M_0=0$ for convenience and use this definition unless explicitly mentioning otherwise
in the rest of the course.
An adapted sequence with finite means is called a \emph{submartingale} if 
$M_n \le \E(M_{n+1}|\F_n)$, and is called a \emph{supermartingale} if 
$M_n \ge \E(M_{n+1}|\F_n)$.  Note that in the case of martingales,
the second condition implies that $\sigma(M_n)$ is a filtration, whereas this is not 
true in the case of submartingales and supermartingales.

\textbf{Notes:}
\begin{enumerate}
\item From the definition of a martingale, we see that $\E(M_p|\F_n) = M_n$ for $p > n$.
\item For a martingale, $\E(M_n)$ is constant.
\item For a submartingale, $\E(M_n)$ is increasing.
\item For a supermartingale, $\E(M_n)$ is decreasing.
\end{enumerate}

An example of a MG is $M_n=S_n - nEX_1$ where $X_i$ is a sequence of i.i.d random variables, 
and $S_n = \sum_{i=1}^{n}X_i$, and $\F_n = \sigma(X_1,\ldots,X_n)$:
\begin{eqnarray*}
\E(M_{n+1}|\F_n) &=& \E(S_{n+1}-(n+1)\E X_1|\F_n) = \E(S_n-n\E X_1|\F_n) + \E(X_{n+1}|\F_n) - \E X_1 \\
 &=& S_n - n\E X_1 + \E X_{n+1} - \E X_1 = M_n. 
\end{eqnarray*}
Also notice that if $X_n=M_n-M_{n-1}$ then $\E(X_n|\F_{n-1}) = 0$. 
Similar results can easily be derived for supermartingales and submartingales.
We will be considering two kinds of results involoving MGs. These are optional stopping theorems, 
maximal inequalities, and convergence theorems.

Martingales and predictable sequences can be thought of in a natural way in gambling systems, 
and gambling systems give us new martingales. 
If $X_n$ is the outcome of the $n^{\text{th}}$ bet in a fair game, and $H_n$ is the 
multiplier that a gambler places for this bet, then her earnings on this bet are $X_n\cdot H_n$. 
Since gamblers can place bets at time $n$ based upon the outcomes at times $1...n-1$, 
$H_n\in \F_{n-1}$ i.e., $H_n$ must be predictable and $S_n$ is a martingale. 
Denoting the gambler's earnings after $n$ bets as a new variable $Y_n$ and taking $X_0=0$, 
we define $(H\cdot S)$ by:
\begin{equation}
H_1\cdot X_1 +\dots H_n\cdot X_n = H_1(S_1-S_0) + \dots + H_n(S_n-S_{n-1}) = (H\cdot S)_n,
\end{equation}
which is called the martingale transform. 

\begin{lemma}[Martingale transform]
If $X_n$ is a $\F_n$-martingale, and $H_n$ is a predictable sequence with $H_nX_n$ integrable for each $n$, then $(H \cdot X)$ is a $\F_n$-martingale.
\end{lemma}
\begin{proof}
$Y_n = (H \cdot X)_n$ is an $\F_n$-martingale if 

\[
\E(Y_n - Y_{n-1} | \F_{n-1}) = \E(H_n(X_n - X_{n-1})|\F_{n-1}) = 0. 
\]
 
 Using $H_n \in \F_{n-1}$, $X_{n-1} \in \F_{n-1}$ and $X_n$ a martingale, we have
\[
 \E(Y_n - Y_{n-1}|\F_{n-1}) = H_n \E (X_n|\F_{n-1}) - H_n X_{n-1} = 0.
 \]
%
%$Y_n = (H\cdot X)_n$ is an $\F_n$-martingale if $E(Y_n-Y_{n-1}|\F_n)=E(H_nX_n|\F_n)=0$. But, if $X_n$ and $X_nH_n$ are integrable, then 
%\begin{equation}
%E(Y_n-Y_{n-1}|\F_n)=E(H_nX_n|\F_n)=H_nE(X_n|\F_n).
%\end{equation}
%We assumed that $X_n$ is integrable, for all $n$.
\end{proof}

Martingales in conjunction with stopping times are a neat way of modeling gamblers strategies. 
A martingale $M_n$ whose differences represent the outcomes at time $n$ may be bet upon by a 
gambler until he stops at some time.
This time is intuitively a stopping time because (by definition of stopping times)
$(T=n)$ is measurable w.r.t
$\F_n$.  This strategy defines the new process $M_{n \wedge T}$.

\begin{theorem}
If $M_n$ is an $\F_n-$martingale and $T$ is a stopping time, then $M_{n \wedge T}$ is also an $\F_n-$martingale.
\end{theorem}
\begin{proof}
Using $H_n=\1_{(T>n-1)}\in F_{n-1}$ in the MG transform formula the result is achieved (note that $H$ is bounded).
\end{proof}

Now, if $M_n$ is a martingale and $T$ is a stopping time bounded by $b$, then 
\begin{equation}
E(M_T) = E(M_{T \wedge b}) = E(M_0)
\end{equation}
In the case of an unbounded stopping time $T$, we have that $M_{T \wedge n}\rightarrow M_T$~a.s.
Hence, if expectations and limits can be swapped as in 
\begin{equation}
\E(M_T) = E\left(\lim_{n \toinf} M_{T \wedge n}\right) = \lim_{n \toinf} E(M_{T \wedge n}) = E(M_0)
\end{equation}
then we can calculate the left hand side.  But, this is not always possible.  For instance, in the case of a random symmetric walk
starting at $S_0=1$ and $T=\inf\{n| S_n=0\}$, we have $ES_T=0$ because $P(T<\infty)=1$. However,  
$1=ES_0=ES_{T \wedge n}$ (!). As we will see later, uniform integrability is enough to justify the swapping
of the expectations and limits.

\bibliographystyle{plain}
\bibliography{../books.bib}

\end{document}